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Digital camera wifh panoraina or mosaic functionality 



FIELD OF THE INVENTION 

The invention relates to an electronics device with a digital camera, and to a 
method of enabling to create a composite picture using a digital camera. 

BACKGROUND ART 

The digital, solid-state cameras are rapidly replacing the traditional analog 
cameras. Prices have come down to a mass-market level and features, unheard of in the 
conventional analog domain, are being added to give the user ever more control over the 
capturing process and the quality of the end-result. CXirrently, digital cameras are 
commercially available that have been designed for connecting to a PC in order to transfer 
image ffles. These ffles can include still image files (e.g., JPEG ffles) as well as motion video 
fttes (e.g., QuickTime files). Such a digital camera captures digital still images and motion 
video segments and stores the images using, e.g.. Flash EPROM memory. Digital cameras 
also are increasingly more fi^quently integrated with other appliances. Devices such as 
mobile phones cunently come equipped with camera chips as a standard feature. Madcet 
reports estimate that in 2002 about 20 million (20M) so-called "camera phones" were sold 
worldwide. The enibedded digital camera enables to instantly share a sn^shot with a remote 
parly. A similar trend of designed-in camera chips has apprared in the PDA market Solid- 
state cameras thus have become a cormnodity in the consumer electronics (CE) environment, 
i.e., the mass martet; wherein brand owners try to distinguish their products by means of 
attractive features providing additional fimctionality . 

An example of the above is discussed in US patent application publication 
20030095193. This reference discloses creating a panoramic digital imag^ by providing a 
digital camera having a memory and vMch is operable in a first mode for producing 
individual still digital images, and in a second mode for capturing a series of overlapping 
digital images to be used in consbucting tihe panoramic digital image. The digital camera is 
mounted on a stand, rotated on the stand through a series of predetermined positions, and 
operated in the second mode to capture flie series of overlapping digital images. The series of 
overlapping digital images conespondiag to the panoramic digital image is stored in a 
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predeteizDined location in the memory, and processed to stitch such images together to 
produce the panoramic digital image. 

SUMMARY OF THE INVENTION 

5 The known camera requires that the xiser accurately follow a specific 

procedure to create a panoramic image. In addition, Ifae camera needs a tripod or other stahle 
stand The inventors propose a user-fiiendlier alternative to the known digital camera for 
creating a panoramic image. In order to capture an image of an ohject, e.g., a scene or a 
building facade, the inventors propose that the user scan the object with a single sweep of the 

10 camera while camera is taking pictures. Upon completing the sweep, the camera stitches the 
pictures taken so as to create a panorama or mosaic picture. 

More specifically, the invention relates to a digital camera having a memory 
and an operational mode wherein the camera takes a sequence of stiU pictures. A next one of 
the pictures in the sequmce is selected for being stored in the memory based on an amount of 

IS overlap regarding a picture content with a previous one of Ifae pictures stored in the memory. 
The camera processes the pictures stored in the memory so as to create a conqiosite picture, 
^•g*9 d panorama or mosaic picture. Preferably, the camera coxcprises a sensor for 
determining information representative of respective relative coordinates of Ifae camera when 
taking respective ones of the pictures during Ifae sweep. Such a sensor conqirises, e.g., an 

20 accelerometer or tachometer firom whose output the camera's positions can be obtained. The 
information can then be used for controlling the creating of the composite picture, e.g., if the 
sweep traces out a serpentine paifa to scan an object that is high as well as wide. Ihe 
information can also be used to d^ermine tiie next picture, for exantple in conjunction with 
information about the estimated distance between the camera and the relevant area of the 

25 scanned object The latter distance can be automatically estimated based on, e.g., tiie focal 
settings of the camera witii auto-focus fimctionality. Preferably, Ifae camera comprises a 
detector for detecting an area in tiie composite picture lacking coverage by the sequmce of 
pictures, e.g., for providing feedbadc to a user of the camera. If the camera has a viewer to let 
the user examine a picture after it has been taken, a reduced version of tiie conoposite picture 

30 created thus fiar can be rendered as welL Any gap is tfaoi easily spotted so that tfae user can 
take an additional picture to cover tfae gap. The user is guided to point tiie cam^ to tfae 
proper area, e.g., by overlaying in the viewer tfae real-time image, i.e., the image resulting 
jBrom capturing at what the camera is being pointed, witii parts of one or more pictures 
already taken that neighbor tfae g^. 
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In a furdier einbodiment of the mvention, the camera as specified above is 
enibedded, or otfaierwise fimctionally integrated, with an electronic apparatos such as a mobile 
phone or pahntop conq)iiter or PDA. 

One of the advantages of having the image processed at the camera is that the 
5 user can ijreview the result afanost instantly in order to deteii^ 

For example, witii a two-dimensional sv^eep the user may have missed an area of the scanned 
object, for which a solution is provided above. Another advantage is that the camera of the 
invention is self-sufficient. Downloading the pictures taken to a PC and post-processing them 
on the PC is not required anymore. In this manner a low-cost digital camera with panorama 

10 and mosaic functionality is provided. 

The invention also relates to a method of creating a conq)osite picture using a 
digital camera. The method comprises taking a sequence of still pictures; and selecting a next 
one of the pictures in the sequence for being stored based on an amount of overlap of a 
picture content with a previous one of the pictures stored Then the pictures stored are 

15 processed so as to create a composite picture. Preferably, the method conqirises determining 
information representative of respective relative coordinates of the camera when taking 
respective ones of the pictures. The information is used for controlling the creating of the 
composite picture using a stitching algorithm and/or to determine the next picture to be 
stored. Preferably, the method comprises identifying an area in the composite picture lacking 

20 coverage by the pictures stored in the memory. 

For completeness, reference is made to WO019S109 (attorney docket 
TW000002) published fer Rob Beuker and Marcel Breeuwer for METHOD AND 
APPARATUS FOR MERGING IMAGES INTO A COMPOSITE IMAGE. This patent 
document relates to an ^iparatus and a method for merging overlapping two-dimensional 

25 (2D) images, \^ch are formed by an image pick-iq) device as projections of a three- 
dimensional (3D) scene. In particular, the merging includes image registration by projective 
transformation of one of the 2D ima^s, the transformation being derived from corresponding 
feature found in both images. In order to achieve improved accuracy and stability, the 
coordinates of the corresponding, feature points are chosen or are translated so that, on 

30 average, the numerical ranges of coordinate values are minimized. The apparatus includes an 
appropriately configured ima^ jirocessor or computist with an attached image acquisition 
device, which in one embodiment, is a diagnostic x-ray. 
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BRIEF DESCRIPTION OF THE DRAWING 

The invention is explained in farther detail, by way of example and with 
reference to the accompanying drawing wherein: 

Fig. 1 is a block diagram of a camera in the invention; and 
5 Figs. 2 and 3 form a flow diagram illustrating a method of the invention; 

Througihout the figures, same reference numerals indicate similar or corresponding features. 

DETAILED EMBODIMENTS 

Fig. 1 is a block diagram of a, preferably hand-held, digital camera 100 

10 according to the inveation. The camera in this example is a still-picture digital camera as 
opposed to a video camera. Camera 100 comprises an image sensor 102, a cache memory 
104, a memory 106, a data processor 108 and a display monitor or viewer 110. Processor 108 
comprises a position/orientation sensor 1 12, an overlap controller 1 14, image stitching 
software 116 and a gap detector 118. 

IS Sensor 102 converts an optical image of an object into digital data and 

ccmiprises a solid-state image-sensing element (not shown), sudbi as a charge-coupled device 
(CCD). Li case a CCD is used, sensor 102 includes an analog-to-digital converter (not 
shown) to convert the CCD's analog ou^ut signals to digital data. Sensor 102 is connected to 
cache 104 that stores the digital data fixim sensor 102. When the user depresses the camera's 

20 shutter button (not shown), processor 108 controls the CCD so as to perform a, so-called, all- 
pixel readout, wherein the data is teniporarily stored in cache 104. From there, the data is 
transferred to memory 106 under control of processor 108. Memory 106 comprises, 6.g., a 
removable memory card or a hard-disk drive (HDD) with a smaU fi)nn-&ctor, etc. Display 
1 10 lets the user view the image as captmed by sensor 102, e.g., while focusing or finding the 

25 proper object, as well as an inoage stored in memory 106. Operation thus far is similar to 
Imown digital cameras. 

Camera 100 of the invention has an operational mode, referred to herein as 
^scan mode", wherein a sequence of images is c^tored that covers an area that is larger than 
can be captured in a single imag^ using the camera's optical components (not shown here). 

30 To this end, the camera has to be moved and/or its orientation changed by the user so as to 
enable taking pictures that together cover the targeted area. Operation is as follows. 

In the scan mode, the user sweeps camera 1 00 along a path for wfaidi the 
accumulated field of view is to cover a certain area. For example, the user sweeps camera 
100 along a path fix>m left to right that in order to capture, e.g., a large articulated steam 
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looomolive on display at a museum, or fiom bottom to top to capture, e.g., a redwood tree. A 
first picture is taken that gets transferred to memory 106. During the sweep camera 100 keeps 
on taking pictures. The reftesh rate of sensor 102 and cache 104 is such that multiple pictures 
canbe taken during the sweep that overlap with the first picture given the typical speed at 
which a human can move his/her arm and wrist holding the camera. Overly controUer 1 14 
determines the overlap of these subsecjuent pictures with the first picture and controls the 
transfer to memory 1 06 of the picture that has a certain minimum, or otherwise desired, 
amount of overlap. This last picture becomes the second picture. Camera 100 keeps taking 
pictures and overlap recognizer 1 14 simflarly selects with respect to the second picture the 
third picture to be transferred to memory 106. and so on to the last picture. The last picture 
relates to the moment at which the scan mode is terminated by means o^ e.g.. the user 
releasing a button (not shown) kept depressed during the sweep and/or stopphig the 
movement of the camera housing. In a simple horizontal sweep the order of the pictures in 
memory 106 is linear. Conventional stitehing algorithms 1 1 6. fcaown in the field of image 
processmg, are run on processor 108 and create a composite picture fiom &e series of 
overlapping pictures available fiom memory 106. 

ContioUcr 1 14 controls the overlap between successive pictures. Controller 
114 is implemented, e.g.. by an algorithm that seeks to matxii pixels of the successive 
pictures m the region of «q,ected overlap. If sensor 1 12 senses camera 1 00 being moved to 
liie right; and prefeably also ifae current speed, and has also mformation about the scanned 

object»s distance fiom the carnera, the expected region of ov«iap can be minhn^ 
angle covered by the camera's lens (not shown). The distance previously mentioned can be 
estimated, based on, e.g., the focal settings. Sensor 1 12 con^irises. e.g., a sin:q,le 
accelerometer. or a tachometer that determines the speed reUtive to the earth's magnetic field 
based on the Hall eflfecl; ete. Altematively, the relative velod^ (virtual pixel displacement in 
a certain direction per unit of time) of pixels in a picture in cache 1 04 relative to the pixels in 
the last picture stored in memory 106 can be determined by means of subjecting subsequent 
pictures to a motion esthnalion algorithm known fiom video processing. As the amount of 
overly depends on the velocity, the overly can be determmed. 

The amount of desired overly is preferably programmable, e.g., made 
automatically depadent on the Ughting conditions, on the high^fiequency conq,onents in the 
picture fliat represent details, or is user-selectable. 

As another exanq)le, the user lets camera 100 trace out a seipentine-like path 
that substantially Ues in a vertical plane to cover, e.g., a fecade of an interesting structure or 
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an object from close by. The oidering of llie pictures taken is a linear one, but the linear order 
is to be mapped onto a two-dimensional array forming the eventual composite picture. 
Stitching now occurs at all four sides of a picture not at tiie perimeter of flie two-dimensional 
area. This means that in order for the stitehing to work, the relative two-dimensional position 
of each of the pictures, successively selected for transferal to memory 106, needs to be 
determined. To this end, camera 108 is equipped with position or orientation sensor 1 12 that 
enables to determine the pafli traced by camera 100 or the focal point of camera 100 during 
the serpentine sweep. Sensor 1 12 comprises, e.g., simple accelerometers (for vertical and for 
horizontal accelerations of tiie camem body and preferably also for determining the relative 
changes in pan and tilt angles) and circuitry to twice integrate over time the acceleration 
values found to determine the patiL The pan or tilt motion can be determined using, e.g., pairs 
of accelerometers tiiat register accelerations in opposite directions. Accordingly, a picture 
selected fer tiansfeal to memory 106 can get associated witii a marie along this pafli 
including tiie relative vertical and horizontal coordinates. This enables to map tiie linear 
1 S ordering onto the two-dimensional air^. 

Figs. 2 and 3 form a flow diagram iUustrating a process 200 according to tiie 
invention- In a step 202, the user depresses a scanhmode button of camera 100, or otiierwise 
causes camera 100 to enter file scan-mode, and starts the sweep. In a step 204, a first picture 
is takai and stored in memory 106. Camera 100 continues to take pictures. The next picture 
20 in cadie 104 is accessedin a step 206 and compared in a step 208 witii the first picture fliat is 
stored in memory 106. In a step 210, it is determined whether tiie picture cunenfly in cache 
104 has an acceptxible overi^ witii the first picture. For exaraple, if flie overiap is large, it 
maybebettertotakeanext pictme, Le., a split second later, tiiat gets loaded mto cache 1 04 
to r^lace tiie previous picture boflferedmcadie 104. -niis he^ to reduce tiie number of 
25 pictures needed for creating tiie con^msite image and makes efficient use of tiie camera's 
onboard storage 106. If tiie overiap is unacceptable, process 200 returns to step 206. If flie 
overlap is acceptable, tiie picture currenfly in cache 104 is transferred to memory 106 in a 
step 212. hi a step 214 it is determined if flie sweep is completed. This is done, fiar exanqile, 
by registering tiiat tiie sweeping motion has stopped or tiiat tiie camera's scaurmode button 
30 has been released. Alternatively, flie user may have set tiie maxhnum number of pirtores to 
be saved to memory 106 in a single sweep, eto. If tiie sweep is still ongoing, process 200 
returns to step 206. If tiie sweep has finished, process 200 proceeds to a step 216 wherem flie 
pictures accumulated in memory 106 are stitehed to create tiie composite hnage, as discussed 
above. In a step 21 8, it is determined whether the. composite image has one or more g^s, i.e.. 
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wfaefher there ace extended areas in ttie composite image wherein pixel data is lacking. If the 
areas are not extended in the sense tha^ e.g., sinq)le interpolation from the area's perimeter 
can fa! in the missing pixels (can be made a sub-routine of step 216), or if there are no gaps, 
process 200 is temiinated in a step 220. If there is a g^ process 200 goes to a step 222, 
wherein the user is guided to the missing area using, e.g., the overlay procedure mentioned 
above. Then, an extra picture is taken covering the missing area and transferred to memory 
106. Thereupon, process 200 returns to step 216 to integrate the extra picture with the 
composite image created thus fer. Note that in the overlay procedure the location of the extra 
picture, that is to be integrated, is aheaify known. 

As an alternative to starting the stitehing together of the pictures in memory 
upon terminating the sweep, processor 108 decides on the coordinates of the pictures stored 
in memory 106 whether some or all of the pictures can akeady be stitched together during the 
sweep. 
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CLAIMS: 



1- A digital camera having a memory and an pperational mode "vdierein: 

the camera takes a sequence of still pictures; 

a next one of the pictures in the sequence is selected for being stored in the 
memory based on an amount of overlap regarding a picture content with a previous one of the 
pictures stored in the memory; and 

the camera inrocesses Ibe pictures stored in the memoxy so as to car^Kte a 
composite picture. 

2. The camera of claim 1, comprising a sensor for determining information 
representative of respective relative coordinates of the camera when taking respective ones of 
the pictures. 

3. The camera of claim 2, wherein the information is used for controlling the 
creating of the composite picture using a stitching algorithm. 

4. Hie camera of claim 2, wherein the information is used to determine tiie next 
picture. 

5. The camera of claim 1, comprising a detector for detecting an area in the 
composite picture lacking coverage by the pictures stored in the memory. 

6. An electronic apparatus with the camera of claim 1 enibedded therein. 

7. A method of oreating a composite picture using a digital camera, the method 
comprising: 

taking a sequmce of still pictures; 

selecting a next one of the pictures in the sequence for being stored based on 
an amount of overlap of a picture content witii a previous one of the pictures stored; and 
processing the pictures stored so as to create a composite picture. 
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8. The method of claim 7, coii5)rismg determining infomiation representative of 

respective relative coordinates of liie camera when taking respective ones of the pictures. 

5 9. The method of claim 8, comprising using the information for controlling the 

creating of the composite picture using a stitching algorithm. 

10. The method of claim 8, using the information to determine the next picture. 

10 11. The method of claim 7, comprising identifying an area in the conqiosite 

picture laddng coverage by the pictures stored in the memory. 
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A digital camera has a m«noiy and a scanrinode. In Uns mode, flie camera 
lakes a sequence of still pictures. Anext one of tiie pictures in the sequence is selected for 
heing stored in Ae memory based on an amount of overlap regarding a picture content with a 
previous one of the pictures stored in the memory. The camera processes the pictures stored 
in Ae memory so as to create a composite picture. 
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